Added workflow execution logic by stevanbz · Pull Request #850 · opensearch-project/alerting

stevanbz · 2023-04-04T21:14:38Z

Issue #, if available:

Description of changes:
This PR contains the changes that enables sequential execution of the workflow delegates. Delegates (monitors) will be executed one by one - and potentially (if the user selected this option) findings of one monitor can be used as inputs for the next monitor in the chain.
Depending of the monitor type, appropriate executor will be called in the same thread in which the workflow execution occurs - meaning that for the time being, parallel monitor execution is not available.
This PR doesn't contain the secure tests for the execution action.
CheckList:
[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

lezzago · 2023-04-06T18:26:35Z

alerting/src/main/kotlin/org/opensearch/alerting/DocumentLevelMonitorRunner.kt

Why are we not doing this all outside of the for loop? No need to get it multiple times.

Are we also requiring the document level monitor to already have its' inputs with the same index for the workflow? Is there some validation that will be for around this in case a user modifies the monitor to have a different index and the the workflow will stop working correctly.

Let's add check during create step to verifying that the monitor have chained findings is querying the same indices as the the monitor whose findings are being used

@lezzago it is getting the documents for each index from the map of index name to list of documents

Why are we not doing this all outside of the for loop? No need to get it multiple times.

You are right. We can move this outside of the loop. During the merging of the changes by mistake I put it here. Makes sense to have it outside of the loop.

Re you second question - workflow is not executing on the index but the monitors that are part of the workflow (delegates) are. There is no such kind of validation right now. @eirsep gave a good point - we can add a validation on create/update workflow.

@eirsep gave good explanation - this map is getting the docIds for the given index from the map of index name and docIds -> this map contains the chained findings (if the chained findings are specified) from previously executed monitor in a chain.

lezzago · 2023-04-06T21:29:01Z

alerting/src/main/kotlin/org/opensearch/alerting/InputService.kt

+                .add(
+                    BoolQueryBuilder()
+                        .must(MatchQueryBuilder("_index", entry.key))
+                        .must(TermsQueryBuilder("_id", entry.value))


Can we optimize this and give the list of doc ids that are part of that index? This seems like it would explode into a very big query that might not be the most optimal.

the max findings would be 10000.

we know definitively that this would be the max size of the terms

@lezzago - not sure that it will explode and I think that we are already doing that -> indexToDocIds is a map in which the key of a map is the index we are searching, while the value is the list of docIds related to the given index. We are using TermsQueryBuilder (TermsQueryBuilder("_id", entry.value)). Maybe I should just renamed the parameter to something more obvious like:

documentIdsPerIndex

findingDocIdsPerIndex

matchingDocIdsPerIndex
@eirsep what do you think?

lezzago · 2023-04-06T21:34:41Z

alerting/src/main/kotlin/org/opensearch/alerting/MonitorMetadataService.kt

What if the document is there, this will throw an exception?

lezzago · 2023-04-06T22:02:42Z

alerting/src/main/kotlin/org/opensearch/alerting/MonitorMetadataService.kt

What does this parameter mean for this function?

This means that the workflow is in dryrun mode for example or it's like a temporary workflow - meaning that it's metadata won't be saved initially - so it skips indexing the metadata

lezzago · 2023-04-06T23:01:03Z

alerting/src/main/kotlin/org/opensearch/alerting/WorkflowService.kt

+            val searchRequest = SearchRequest()
+                .source(
+                    SearchSourceBuilder()
+                        .query(bqb)
+                        .version(true)
+                        .seqNoAndPrimaryTerm(true)
+                        .size(size)
+                )
+                .indices(ScheduledJob.SCHEDULED_JOBS_INDEX)


Should we add pagination logic? Do we have a limit on the number of monitors can be part of a workflow also?

We don't have the limit of the monitors that can be part of the workflow. Maybe it makes sense to introduce this kind of validation. @eirsep what do you think about this idea?
If we introduce limitation, depending on the size, maybe we won't need the pagination logic.
@lezzago - my thinking is that hardly we're gonna reach the maximum (since we are passing the monitor ids that belong to a workflow, and I can hardly imagine that we are calling this function for maximum terms query)

lezzago · 2023-04-06T23:06:59Z

alerting/src/main/kotlin/org/opensearch/alerting/action/ExecuteWorkflowAction.kt

This should be named as cluster:admin/opensearch/alerting/workflow/execute

lezzago · 2023-04-06T23:16:00Z

alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportExecuteWorkflowAction.kt

+                val workflow = when (user?.name.isNullOrEmpty()) {
+                    true -> execWorkflowRequest.workflow as Workflow
+                    false -> (execWorkflowRequest.workflow as Workflow).copy(user = user)
+                }
+                executeWorkflow(workflow)


We have additional logic needed for Document level monitors. This doesn't take care of that.
reference: https://github.com/opensearch-project/alerting/blob/main/alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportExecuteMonitorAction.kt#L127

Ah I see. Very very good point.

I think we don't need to do the same thing when executing the workflow.
why? Because workflow in itself will contain the list of delegate monitor ids (which are needed to exist - if not, they won't be executed). meaning that, if some of the monitorIds are document level monitors (which let's say exist), then it would mean that those document level monitors already have docQuery index initialized
The workflows only work with pre-created monitors

lezzago · 2023-04-07T00:01:57Z

alerting/src/main/kotlin/org/opensearch/alerting/workflow/CompositeWorkflowRunner.kt

Should we use UTC time for this? This would help with debugging purposes.

lezzago · 2023-04-07T00:03:14Z

alerting/src/main/kotlin/org/opensearch/alerting/workflow/CompositeWorkflowRunner.kt

This should be more descriptive and mention it failed to get the monitors.

plz never log e.message

it swallows the entire stacktrace
the format should be log.error("Message", e)

plz never log e.message

it swallows the entire stacktrace the format should be log.error("Message", e)

Yeah - I agree. Copy-pasted from documentMonitorRunner. Makes sense

lezzago · 2023-04-07T00:06:51Z

alerting/src/main/resources/org/opensearch/alerting/alerts/finding_mapping.json

      "type": "long"
+    },
+    "execution_id": {
+      "type": "keyword"


We need to increment the schema version whenever we update the schema for a release.

eirsep · 2023-04-07T03:49:09Z

alerting/src/main/kotlin/org/opensearch/alerting/MonitorMetadataService.kt

can we have workflowmetadataservice

We can have - I thought that it makes sense to be on one place - but yeah since we already separated the workflow in completely separate stream why not.

eirsep · 2023-04-07T03:50:10Z

core/src/main/resources/mappings/scheduled-jobs.json

256 is a small number
do we need this for a list?

Yes it is. So to set to 1000?

eirsep · 2023-04-07T03:50:31Z

alerting/src/test/kotlin/org/opensearch/alerting/TestHelpers.kt

 val TEST_NON_HR_INDEX = "not_hr_data"
 val TEST_HR_ROLE = "hr_role"
 val TEST_HR_BACKEND_ROLE = "HR"
-


NIT: revert this

eirsep · 2023-04-07T03:52:06Z

alerting/src/main/kotlin/org/opensearch/alerting/workflow/CompositeWorkflowRunner.kt

+        monitors: List<Monitor>,
+        workflow: Workflow,
+    ) {
+        if (delegates.size != monitors.size) {


let's add error log also

eirsep · 2023-04-07T03:53:53Z

alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportExecuteWorkflowAction.kt

+                        override fun onResponse(response: GetResponse) {
+                            if (!response.isExists) {
+                                actionListener.onFailure(
+                                    AlertingException.wrap(


eirsep · 2023-04-07T03:53:59Z

alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportExecuteWorkflowAction.kt

+                        }
+
+                        override fun onFailure(t: Exception) {
+                            actionListener.onFailure(AlertingException.wrap(t))


eirsep · 2023-04-07T03:54:56Z

alerting/src/main/kotlin/org/opensearch/alerting/model/WorkflowRunResult.kt

+    val executionStartTime: Instant,
+    val executionEndTime: Instant? = null,
+    val executionId: String,
+    val error: Exception? = null


should we make this a list?

Well, I think we discussed to remember only the last error if happened. We had a discussion on previous version of the PR, and that's what we agreed. So the last execution error will be handled like here
Also, bear in mind that the
val workflowRunResult: List<MonitorRunResult<*>> = mutableListOf(),
can contain the errors if the error happens;

Like we are doing here for doc level monitors, while bucket level monitor runner is not handling the error on the same way - ie. the code is not wrapped with try-catch. Hm need to think

eirsep · 2023-04-07T03:57:09Z

alerting/src/main/kotlin/org/opensearch/alerting/WorkflowService.kt

log.error("message",e)

eirsep · 2023-04-07T03:59:18Z

alerting/src/main/kotlin/org/opensearch/alerting/workflow/CompositeWorkflowRunner.kt

log.error("",e)

eirsep · 2023-04-07T04:00:13Z

alerting/src/main/kotlin/org/opensearch/alerting/workflow/CompositeWorkflowRunner.kt

i hope we are not updating monitor metadata if there was an error in the workflow

Check the comment above - need to think how and when to cancel workflow metadata update. In any case, the workflow meatadata are not used like for example metadata in doc level monitors (and their lastRunContext)

lezzago · 2023-04-12T16:58:04Z

alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportIndexWorkflowAction.kt

+                if (!delegateMonitorIndices.equalsIgnoreOrder(chainedMonitorIndices)) {
+                    throw AlertingException.wrap(IllegalArgumentException("Delegate monitor and it's chained finding monitor must query the same indices"))
+                }


Should we include this logic in the TransportIndexMonitorAction class? This way its to ensure users don't accidentally update those monitors to change the indices and cause the workflow to get messed up? Though the other issue is if they want to update the index for all the monitors, they would have to delete the workflow and update the monitors and then create a new workflow. I am open to leaving it as is, but if we can then have some messaging informing the user of this, that would be nice.

Thanks for pointing out this corner cause of updating all monitors failing sucha check.
This is a little tricky.
For security analytics this use case will never have to be handled as all monitors will have same indices queried always.
Let's add a FIXME and defer this to be handled when we have Alerting plugin using workflows in future.
I will propose a solution for this by then.

lezzago · 2023-04-12T17:24:08Z

alerting/src/main/kotlin/org/opensearch/alerting/workflow/CompositeWorkflowRunner.kt

why are we using the workflowMetadata ID now instead of workflow ID?

can we have two separate fields with both

Good question. Will try to answer you here - if you have any doubts you can reach me directly:
@eirsep - yes and I was thinking the same.

WorkflowMetadataId - is either workflowId or some random uuid generated depending if the workflow is executed in dry run mode or not (code sample for generating the id:
if (isTempWorkflow) "${LocalDateTime.now(ZoneOffset.UTC)}${UUID.randomUUID()}" else workflow.id
)
WorkflowContext.workflowId is used when generating the document level monitor metadata - in order not to have collision when executing same monitor that belongs to two different workflows (because doc level monitors are keeping the history of lastRunContext which is extracted from doc level monitor metadata). Also, when running the workflow in dry run mode, we need to load all the documents from scratch - and not to use the lastRunContext of the docLevelMonitors.

That's why I set the workflowMetadata.id -> I would say that Sashank's comment gives the answer:
Will extend the WorkflowContext to have also workflowMetadata.id and this id will be used in order to generate the doc level delegate monitor metadata

lezzago · 2023-04-12T17:34:26Z

core/src/main/resources/mappings/scheduled-jobs.json

            "keyword": {
              "type": "keyword",
-              "ignore_above": 256
+              "ignore_above": 1000


Why does this need to be changed to 1000? Can there be some sort of comment around it?

+1
let's also add a check in indexworkflow that we limit the number of monitors in sequence to 25 for now.
We can revisit that later.

Can we add some comment the here to revisit and to have it go back to 256 possibly?

We auto-generate monitor ids and Auto-generated ids are of 20 characters length.

@stevanbz plz add a comment that monitor_id is typically 20 characters long and hence monitor_ids field would need to be 1000 to be able to accommodate 50 delegate monitor in a workflow

But we can set the limit of delegate monitors between 20-25 to account for performance implications of a workflow executeion

So, the monitor id generated has size of 20. If we have 25 monitors, that means that the limit can be 500.
According to agreement with @lezzago, we are keeping as it is

Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com>

…lse while the metadata alerady exist Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com>

Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com>

…evel monitor findings Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com>

…ing the workflow Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com>

lezzago · 2023-04-19T16:12:06Z

alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportIndexWorkflowAction.kt

        val searchRequest = SearchRequest(SCHEDULED_JOBS_INDEX).source(searchSource)

-        if (user != null && !isAdmin(user) && filterByEnabled) {
+        if (user != null && filterByEnabled) {


Why are we removing the !isAdmin(user) piece? Admin users bypass the filter by check.

lezzago · 2023-04-19T16:19:41Z

approving, but we will need to follow up with admins bypassing the filter by check

* Added workflow execution logic Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Adjusted code according to comments Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Updated version of the findings json Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Updating the workflow metadata in the case of updating flag set to false while the metadata alerady exist Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Added logging for workflow metadata update Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Added Rest Execute Workflow action Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Extended workflow context with workflowMetadataId. Adjusted the doc level monitor findings Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Updated conditions for unstashing the context when indexing and deleting the workflow Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> --------- Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com>

* Added layer for creating and updating the workflow (#831) * Renamed chainedFindings to chainedMonitorFindings * Removed unecessary mappings from workflow definition * Improved logging when saving the workflows * Added a workflow id in response * Added role check and index access once the workflow is being created * Updated mappings for the workflow --------- Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Fixed xContent dependencies due to OSCore changes (#839) Signed-off-by: Angie Zhang <langelzh@amazon.com> * Dependency fix (#846) Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Refactored workflowIndexing validation - removed coroutine and contex… (#857) * Refactored workflowIndexing validation - removed coroutine and context client context lost Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * refactored getting the workflows Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Changed the logic according to secure test findings Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * [Backport 2.x] Notification security fix (#861) (#863) * Notification security fix (#852) * added injecting whole user object in threadContext before calling notification APIs so that backend roles are available to notification plugin * compile fix * refactored user_info injection to use InjectSecurity * ktlint fix --------- (cherry picked from commit e0b7a5a) * remove unneeded import --------- Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Petar Dzepina <petar.dzepina@gmail.com> Co-authored-by: Ashish Agrawal <ashisagr@amazon.com> * Stashed user together with it's roles Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> --------- Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Petar Dzepina <petar.dzepina@gmail.com> Co-authored-by: Ashish Agrawal <ashisagr@amazon.com> * Added workflow execution logic (#850) * Added workflow execution logic Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Adjusted code according to comments Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Updated version of the findings json Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Updating the workflow metadata in the case of updating flag set to false while the metadata alerady exist Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Added logging for workflow metadata update Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Added Rest Execute Workflow action Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Extended workflow context with workflowMetadataId. Adjusted the doc level monitor findings Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Updated conditions for unstashing the context when indexing and deleting the workflow Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> --------- Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Added fix when executing the workflow and when chained findings index… (#890) Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Fixed deleting monitor workflow metadata (#882) * Fixed deleting monitor metadata and workflow metadata. Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix monitor metadata error from conflict resolution Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * remove unused import Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * remove rest execute workflow action Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * increment schema version for findings mapping json Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> Signed-off-by: Angie Zhang <langelzh@amazon.com> Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Co-authored-by: Stevan Buzejic <buzejic.stevan@gmail.com> Co-authored-by: Angie Zhang <langelzh@amazon.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Petar Dzepina <petar.dzepina@gmail.com> Co-authored-by: Ashish Agrawal <ashisagr@amazon.com>

(cherry picked from commit e0b7a5a) * remove unneeded import --------- Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Petar Dzepina <petar.dzepina@gmail.com> Co-authored-by: Ashish Agrawal <ashisagr@amazon.com> * Stashed user together with it's roles Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> --------- Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Petar Dzepina <petar.dzepina@gmail.com> Co-authored-by: Ashish Agrawal <ashisagr@amazon.com> * Added workflow execution logic (opensearch-project#850) * Added workflow execution logic Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Adjusted code according to comments Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Updated version of the findings json Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Updating the workflow metadata in the case of updating flag set to false while the metadata alerady exist Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Added logging for workflow metadata update Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Added Rest Execute Workflow action Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Extended workflow context with workflowMetadataId. Adjusted the doc level monitor findings Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Updated conditions for unstashing the context when indexing and deleting the workflow Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> --------- Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Added fix when executing the workflow and when chained findings index… (opensearch-project#890) Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> * Fixed deleting monitor workflow metadata (opensearch-project#882) * Fixed deleting monitor metadata and workflow metadata. Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * fix monitor metadata error from conflict resolution Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * remove unused import Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * remove rest execute workflow action Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> * increment schema version for findings mapping json Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> --------- Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> Signed-off-by: Angie Zhang <langelzh@amazon.com> Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Co-authored-by: Stevan Buzejic <buzejic.stevan@gmail.com> Co-authored-by: Angie Zhang <langelzh@amazon.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Petar Dzepina <petar.dzepina@gmail.com> Co-authored-by: Ashish Agrawal <ashisagr@amazon.com>

(cherry picked from commit e0b7a5a) * remove unneeded import --------- * Stashed user together with it's roles --------- * Added workflow execution logic (#850) * Added workflow execution logic * Adjusted code according to comments * Updated version of the findings json * Updating the workflow metadata in the case of updating flag set to false while the metadata alerady exist * Added logging for workflow metadata update * Added Rest Execute Workflow action * Extended workflow context with workflowMetadataId. Adjusted the doc level monitor findings * Updated conditions for unstashing the context when indexing and deleting the workflow --------- * Added fix when executing the workflow and when chained findings index… (#890) * Fixed deleting monitor workflow metadata (#882) * Fixed deleting monitor metadata and workflow metadata. * fix monitor metadata error from conflict resolution * remove unused import * remove rest execute workflow action * increment schema version for findings mapping json --------- Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> Signed-off-by: Angie Zhang <langelzh@amazon.com> Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Co-authored-by: Stevan Buzejic <buzejic.stevan@gmail.com> Co-authored-by: Angie Zhang <langelzh@amazon.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Petar Dzepina <petar.dzepina@gmail.com> Co-authored-by: Ashish Agrawal <ashisagr@amazon.com>

(cherry picked from commit e0b7a5a) * remove unneeded import --------- * Stashed user together with it's roles --------- * Added workflow execution logic (#850) * Added workflow execution logic * Adjusted code according to comments * Updated version of the findings json * Updating the workflow metadata in the case of updating flag set to false while the metadata alerady exist * Added logging for workflow metadata update * Added Rest Execute Workflow action * Extended workflow context with workflowMetadataId. Adjusted the doc level monitor findings * Updated conditions for unstashing the context when indexing and deleting the workflow --------- * Added fix when executing the workflow and when chained findings index… (#890) * Fixed deleting monitor workflow metadata (#882) * Fixed deleting monitor metadata and workflow metadata. * fix monitor metadata error from conflict resolution * remove unused import * remove rest execute workflow action * increment schema version for findings mapping json --------- Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> Signed-off-by: Angie Zhang <langelzh@amazon.com> Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Co-authored-by: Stevan Buzejic <buzejic.stevan@gmail.com> Co-authored-by: Angie Zhang <langelzh@amazon.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Petar Dzepina <petar.dzepina@gmail.com> Co-authored-by: Ashish Agrawal <ashisagr@amazon.com> (cherry picked from commit 171cd2e)

(cherry picked from commit e0b7a5a) * remove unneeded import --------- * Stashed user together with it's roles --------- * Added workflow execution logic (#850) * Added workflow execution logic * Adjusted code according to comments * Updated version of the findings json * Updating the workflow metadata in the case of updating flag set to false while the metadata alerady exist * Added logging for workflow metadata update * Added Rest Execute Workflow action * Extended workflow context with workflowMetadataId. Adjusted the doc level monitor findings * Updated conditions for unstashing the context when indexing and deleting the workflow --------- * Added fix when executing the workflow and when chained findings index… (#890) * Fixed deleting monitor workflow metadata (#882) * Fixed deleting monitor metadata and workflow metadata. * fix monitor metadata error from conflict resolution * remove unused import * remove rest execute workflow action * increment schema version for findings mapping json --------- Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com> Signed-off-by: Angie Zhang <langelzh@amazon.com> Signed-off-by: Ashish Agrawal <ashisagr@amazon.com> Signed-off-by: Surya Sashank Nistala <snistala@amazon.com> Co-authored-by: Stevan Buzejic <buzejic.stevan@gmail.com> Co-authored-by: Angie Zhang <langelzh@amazon.com> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Petar Dzepina <petar.dzepina@gmail.com> Co-authored-by: Ashish Agrawal <ashisagr@amazon.com> (cherry picked from commit 171cd2e) Co-authored-by: Surya Sashank Nistala <snistala@amazon.com>

stevanbz requested a review from a team April 4, 2023 21:14

lezzago requested changes Apr 7, 2023

View reviewed changes

eirsep reviewed Apr 7, 2023

View reviewed changes

alerting/src/main/kotlin/org/opensearch/alerting/WorkflowService.kt Outdated

Copy link

Member

eirsep Apr 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

log.error("message",e)

eirsep reviewed Apr 7, 2023

View reviewed changes

alerting/src/main/kotlin/org/opensearch/alerting/workflow/CompositeWorkflowRunner.kt Outdated

Copy link

Member

eirsep Apr 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

log.error("",e)

eirsep reviewed Apr 7, 2023

View reviewed changes

lezzago reviewed Apr 12, 2023

View reviewed changes

stevanbz force-pushed the feature/composite-monitors-workflow-execution branch from d1f86e8 to aa6b611 Compare April 12, 2023 20:09

stevanbz added 7 commits April 18, 2023 18:22

Added workflow execution logic

03ab36f

Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com>

Adjusted code according to comments

a38ff87

Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com>

Updated version of the findings json

b79e002

Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com>

Updating the workflow metadata in the case of updating flag set to fa…

35742fc

…lse while the metadata alerady exist Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com>

Added logging for workflow metadata update

e8d447a

Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com>

Added Rest Execute Workflow action

9467ef2

Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com>

Extended workflow context with workflowMetadataId. Adjusted the doc l…

e418851

…evel monitor findings Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com>

eirsep approved these changes Apr 18, 2023

View reviewed changes

stevanbz force-pushed the feature/composite-monitors-workflow-execution branch from aa6b611 to e418851 Compare April 18, 2023 18:39

Updated conditions for unstashing the context when indexing and delet…

b303435

…ing the workflow Signed-off-by: Stevan Buzejic <buzejic.stevan@gmail.com>

stevanbz force-pushed the feature/composite-monitors-workflow-execution branch from 0b542f1 to b303435 Compare April 19, 2023 07:14

lezzago reviewed Apr 19, 2023

View reviewed changes

lezzago approved these changes Apr 19, 2023

View reviewed changes

eirsep merged commit ac1640d into opensearch-project:feature/composite-monitors Apr 19, 2023

eirsep mentioned this pull request May 24, 2023

Adds transport layer actions for CRUD workflows #932

Closed

eirsep mentioned this pull request May 24, 2023

Adds transport layer actions for CRUD workflows #934

Merged

Conversation

stevanbz commented Apr 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eirsep Apr 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eirsep Apr 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevanbz Apr 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevanbz Apr 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevanbz Apr 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevanbz Apr 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevanbz commented Apr 4, 2023 •

edited

Loading

eirsep Apr 7, 2023 •

edited

Loading

eirsep Apr 7, 2023 •

edited

Loading

stevanbz Apr 7, 2023 •

edited

Loading

stevanbz Apr 7, 2023 •

edited

Loading

stevanbz Apr 7, 2023 •

edited

Loading

stevanbz Apr 7, 2023 •

edited

Loading

stevanbz Apr 7, 2023 •

edited

Loading